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ABSTRACT ; ^ • 

The general emphasis of Project LONGSTEP was on t 
.identification of changed in student achievement that occur as a 
result of exposure to intensive educational innovation. The purpo 
of ^he analyses conducted for this supplement to Volume I of the 
Project LONGSTEP final report was (1) to ascertain if the trends 
observed^ and discussed in Volume I with respect to students in gr 
1, a, and 6 during the 1970-71 school year were represe'ntative of 
trends shown by all ajfalysis samples and (2) to. compare results a 
determ^ine if other meaningful trends across cohorts were ]pte sent, 
primary research hypothesis, that substantial gains in achievemen 
test performance are positively associated with innovative e.mphas 
has not been supported in any general way by the analysis of Proj 
LONGSTEP »s data. These results, based on a global analysis of tre 
across reading and arithmetic analysis samples and grades, tend t 
confirm the findings^ reported in Volume I for students in grades 
and 6. "Data Collection Instruments and Guidelines" developed 
Project LONGSTEP referenced in Vol. I, Chapter II, Section C, wil 
accessioned TH 005 98? in RIEMAY77. (RC) 
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.. INTRODUCTION TO. PROJECT LONGSTEP' 

Educators and noneducators alike haye shown a growing awareness of 

the lack of — and need for — evidence as'to whether/ or not innovative^u- ' 

* cational practices are* indeed better than the morie traditional approaches. - 

'In response to this need, the U. S.' Offi^e^f Education i-n 1969 awarded a 
t 

contract to 'the Mnericah^ Institutes, for Research to develop a^ design, f^ 
a^ study of the effectiveness of .highly intensive, innovative educational' 
practices on. students^ in grades 1 through 12.- The general emphasis of\ ^ 
the .resulting Project LQNGSTEP (the Longitudinal Study of Ec^ucational 
Pri^ctices) was on the identification of- changes in student achievement 
that occur as a result of intensive educational • innovation, "intensive 
innovation 'ma^aning the implementation of a new program encompassing a 
significant proportion of students, entailing a major alteratioh of School 
procedures, and involving a high investment of"" re^sourcefe . >^ 

Si^egific objectives^^of-Froject LpNGSTEP were to design a sy$tem to 
study the characteristics underlying innovative educational approaches; 
to establish a • large-scale data base of program characteristics and 'student 
outcomes for a se4ax:i: sample of edilcational p/ograms involving intensive 
and^ highly innovative Education practices; to determine longitudinally 
the impact of such innovation upoif student performance and attitudes; and 
to attempt to identify the dimensions of the components tl^iat exhibited 
the greatest impact on student outcomes- « % 

A complete discussion of the project design and' data collection, as 
well as the .analytical methods and findings for thtee cohorts, is con-^ 
tained in Volume I of the kinal^ report (Coles, Chalupsky, Everett, '"^t 
^aycoft, Rodabaugh, and Danof f , 1976)'- ^This supplementary report has been 
prepared with the expectation that the readeiV is familiar with VolimfeNl. 
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II. OVERVIEW OF THIS REPORT SUPPLEMENT 

, Volume I of the final. report of the Longitudinal Study of ^Educational 
Practiv:es (Project LpNGSTEP) was written to provide a detailed discussion 
of (1> the .sfudy's overall objectives, design, data' collection procedures, 
instrument design and scaling, and data base and (2) the inethods used and 
the f indings^obtain^d in an intensive analysis oythe treatment and teacher 
correlates of reading, -language and arithmetic achievement for three 
cohorts^ of students— Cohorts 1, 4 and 6 . This Volume I Supplement, on 
the other hand, describes and integrates the study's overall results 
obtained by meaixs of a more global look at the reading and arithmetic 
achievement of all cohorts of students who participated in Project LONGSTEP^ 
The primary jobjectives of this njore general presentation of results are 

• to fiis&uss the. f indings'^ and the*^ conclusions based on the 
analysescof all cohorts and, where appropriate , compare 
such finciings with those obtained for Cphorts 1, 4 and 

6 (as described in Volume I) \ 

• to compare the educational growth of different cohorts 

of students when they were at similar grade levels (during 
different school years) or the growth of the same group 
> ■ of students in two consecutive school years. * ^ . 



^'Cohgrt" is a term that is used to identify a given group of students 
who followed the same grade progr^sion during the three years that the 
study was implemented. Cohorts are labeled by the grade level of that 
group of students during Y^r 1 of the study, the 1970-71 school ye su:.. 
^ Thus, Cohort 1 refers to all those students who were first-graders during 
the 1970-71 'school year or who were not present in the -sample during Year.l 
but who"^ would have been first-graders at that time be^cause they were 
second-graders in Year '2 or third-graders in Year'' 3. Similarly, Cohort 4 ' 
would identify the students who were in the fourth grade ,in 1970^1. The 
term "cohort*«was. utilized throughout the Project LONGSTEP report to iden- 
V^tify styd^nt groups because the study's longitudinal design meant that a - 
;46iven group of .students would be members of three different 'grades, the 
"^''particular grade depending on the .particular .school year. Table 1 reviews 
' 'the grade membership of each cohort of studei^s present in Project LONGSTEP 
'during each of ' the study's three years of data collection. 
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* Gr^de Membership for. 

Students in All-Cohorts by SAiopl Year. 
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The findings presented in this supplement, by design, were not based 
on the entire sequence of analyses that were used to examine intensively 
the achievement of st/ddents in Cohorts 1, 4 and 6., Rather, a sx^set of ' 
these approaches wa^ used to <?permit a more general examination and evalua- 
tion of Project LO^GSTEP's primary research hypothesis that, substantial 
educational growth is positively associated with greater emphasis ^on inno- 
vation. (Due tJ the similarity of "the Reading and Language 'results 
reported in Vo^W i, this supplemental report focuses only„ on' reading and 
arithmetic ach;ievementO The remainder of this report is organised into 
, two sectdons/ the ^irst^f which briefly reviews the :^articular analytic 
approach uD6n which the findings -and conclusions repotted. here are b^sed. 
The last section summarizes the findings and contains -ari^ o.^fer view of 
trends across cohorts and cognitive outcomes. ^ * 

III. ANALYTIC APPROACH 

. / ., ' . ■ ■ ■ . . " ■ , 

r. /Analytic procedures implemented for this .supplement to Volume I 
noi^ded most of the methods used to evaluate the impact of innovative 
enlphasis on reading, language and arithmetic achievement in Coho/ts, 1, 4" ' 
^d 6. These are reviewed in the remainaer of this section-in terms of 
^/the questions they were designed to answer.. .Discussion of 'issues and 
i_ procedures is purposely brief since most were treated in detail in Chapter 
iV of Volume I. ' . * , • . y " 

1. What general analytic approach was V:hosen to assess growth? 

As with the Cohort J., 4 and 6 analyses conducted for Volume I, two 
pretestyposttest analyses were iitijplemented on each cohort's data. The' 
Year 1/Year 2 analyses used" the Spring 1972 test 1^ a^'Posttest and the 
Spring 1971 test as a pretest. ^These analyses have also been called the ^ 
Spring 1972 (abbreviated as SP72) analyses throughout Volume I. The 
Spring^ 1973 (or SP73) analyses examined growth between Year 2 and Year 3, 
\ that .is;' betweeti Spring 1972 (the pretest) and Spring 1973 (the posttest). 
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2. How was ther potential analysis sample defined? . - 

• Membership, in analysis samples i/as defined by the criteria discussed 
fully in Volume I, In general, the students contaitied in a given pretest/ 
^osttest analysis sample were those who were present in the Project LONGSTEP 
sample during both the pretest and post test school y^rs and who^ad no 
missing data with respect 'to the key analysis variables, examined "for this 
supplementary report. (These variables are a subset^of thos^ examined 
previously in Volume- I and vate' listed in Question 4.) 

3. How many students were deleted from each aflalysis sample becalise of 
missipg data or maximum/ minimum test scores? ' 

Table 2 shows the number of students in each analysis ^ample and the 
number of students deleted from analysis because they had some form of 
missing data or because they obtained a near perfect score ("topouts") or 
n^ar zero scof^^-bottomouts") on the pretest and/or posttest. Due to the 
complex logical^d statistical problems in^;ST^ in developing reasonable" 
est-dmates of the impact of missing data on inferences (described more 
.fully in Volume I), cost and 'time constraint!* did not permit special', 
intensive analyses to be 'implemented on deleted stu^nts, (Missing data 

and.test floor/ceiling effects, however, have been 'discussed more fuli|^^0^ 
Volume I, Chapters IV ..and V.) f ' ' ^ ' '' ' . : 

4, Which key variables were examined in these supplementary analyses? " 

Although all of the individual key. analysis variables discussed in 
Volume I have been included in the Cables and matrices of descriptive 
statisti^:s accompanying tl>is supplementary report^, the findings to be dis- A 
cussed in Section lll focus only on the relationship o^f intensity of inno- 
vative emphaais to growth in achievement. Thq measures of primary inter- ' 
est in the analyses reported here include 

/ •' , . ' ' • - . ■ • 

• pqsctest achievement, measured by the CTBS Reading Total ' . 

or Arithmetic Total Expanded Scale Score correAed for 
, site differences in.'time of testing (see Volume I) ^ 

• home background, measured by the Socioeconomic Status (SES) 
.of e^ch student's home en\d:ronment 
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• , ■ V /. TABLE 2 

' N,uinbers'of Students ' in - Each Analysis Sample for All Cohorts 



Cohort 


Depejident " 
Variable' 




Spring;L972 Posttest Analy^^is 




Spring 19^73 Posttest 


Analysi 


s 


To til 


, Complete" 
■ Dat4 


'Missing 
Data* 


„ ^ Bottom- 

. Topoutj 

, outs 


Total 


Complete 
Data 


Missing 
Data* 


Topouts 


Bottom- 
outs 


0 


Reading 






NA 




982 


674 


308 








Arithipetic 








9^2 


640 


342 




- 


= \ 


Reading ^ 


m. 
'I 


791 


184 




1049 


767 


282 








Arithmetic 


953 


752 


201 




1012 


741 


259 


' 12 


- 




Rfeadirig 


'999 


805 


189 


5 


1057 


757 


278 


22 






Arithmetic 


1062. 


79.3 

I " 


■'209 




1048 


699 


.297 


52 






Read-in^ 


232A 


1879 


^ 400 


• 45 


2422 


1905 


476 


35^ 


6 




(Arithmetic 


, 2293 


" 1808 


368 


. .117- 


2409 


1808 


601' 








Reading 


2535 


1952 


529 ' 


V 

54 


2439 


1925 ■ 


499 


' 15 






Arittimetic 


.2537 


1943 


527 ■ 


67 . - 


2521 


1964 


509 ' 


48 






^Reading 


2337 


1916 


396 


2^. - 


2453 


-^^794 


567 


.21 


71 ". 




Arithnjj^tic 


232A 


18A5 


423 


38 18 


2442 


1804 


554 




45 


* 6 . 


Reading 


2183 


1520 


, 647 


? 16 


1968 


1552 


405 


11 






Arithmetic 


. 2111 


1A43 


614 


^ 54 


1960 


1505 


393 


62 




7 


Reading 


2116 


1484 • 


^ 613 


^19 - ' 


2096 


1241 


720 


13 


12( 




Arithmetic 


2106 


1299 


^ 635 


56 116 


2079 


1172 


738 


45, 


^ 124 



13 



^ (continued) 
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TABLE 2 (continued) 



Cohort 


^ Dependent 
'Variable 


Total 


Spring 1972 Posttest Analysis 
Coir.plete Missing _ 
Data ■ - Data 


Bottom- 
outs 


.Total 


Spring 1973 Posttest Analysis 

Complete Missing. _ 
J Topouts 
Data Data ^ 


Bottom- 
outs 


8 ' 


Reading" 


2076 


1406 


660 


10 




1918 


1238 


749 


6 


163 




Arithmetic 


2060 


1295 


505 


47 


213 


1917 


1069 


703 


30 


.115 • 


*9 


Reading 


1161 


699 


457 


5 




1430 


846 


449 


14 


121 




Arithmetic 


1159 


.- 554 


527 


. 15 


63 


1420 


464 


850 


42 


64 


10 


Reading 


1443 


850 


573 


^ 20 




" 1290 


676 


492 


23 


99 




Arithmetic 


1419 


502 


821 


50 


46 ■ 


. 1282 


^ 215 


999 


,48 


20 



' Students not enrolled in reading or arithmetic classes (be cause' they may have been electives, 
especially at the higher grade levels) would have had missing treatment and teacher data and 
there fore, would have been classified as studS'nts with missing data. 
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0^ ^ . ■ 

• initial achievement, status or pretest, measur^diby the 

^ appropriate subtest '^score obtained by each student dur.ing 
the previous school year (and corrected for site differ- 
ences in. time of testing) '<-^ " • 

• , Level of Innovation, a measure of general emphasis, qn 

/ innovation' equal to the^sum.of ten, key indices pf sgbool 



practices and procedures , each index scaled »so.%iat a - 
higher score would reflect greater judged prJ^gram- level 
emphas'ie on innovation^ - " 

m Numbet '©.p per Eray, measured separately with * , 

respect :^'0^^^ £(moxint ipf class time^ipent per d^y on , " . . 
lang^uage ^rts or mathematics ^ct jftvi!j|ia^s" 

. , y ^ ' > ■' ; . ' . . • \ • ■. ■ . <f^^, 

• Teacytiojg QualgLf icati6ns , measured b^y^ the a'yferage Teaching* . 

Qualifjidlpit Ions \^ scale '3 tare (cifi "inde^' cTf. te4:cher education 

and e^^^^ri^jyt) of a given st-ndent^^s'' teachers for each ^ 

subject matf^t area separately (i. e. , sdparate^.y for each 

student's language .arts ^j^^jnd^^to teachers). 

An additional outcome, measure eqUalto.;the part of each student's 
posttest score that could not <be linearl^' predicted frqm his/her pretest 
and SES was also computed. As noted in Volume I (Chapter IV), the analy- 
sis of the relationships between such a residualized posttest score and 
a set of predictor variables is based on a different educational and sta- 
tistical model than an analysis of the relationships b^tt^een posttest and 
a set of residualized predictor variables. This variable was included 
here so that the association between inndvative emphasis and achievement^ 
growth could be examined in a slightly different way to minimize the 
possibility that conclusions regarding impact would be highly method 
dependent . 



Degree of Individualization (DI) , a scale based on a subset of the items 
used to form Level of Innovation (LI), was not intensively analyzed as it 
was in Volume I because of time and cost . uiiitations and the redundancy in 
the two super^les. Nevertheless, DI- is included in the various correla- 
tion matrices gKuaEated for the report and reproduced in the supporting 
appendices. 

16 . 
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How .large were the pretest/posttest gains in each analysis' sample and 

how do they compare across cohorts and grade levels? 

Volume I which looked at data obtained from Cohorts 1, 4 and 6 showed 
that the magnitude of the^ yearly gains in achievement (expressed in stan- 
dard deviation units-^Hi^ Volume I, Equation V-1 on p. 145) decreased at 
each hijher grade ^^1^ -^Although such analyses are not central to the 
objectives- of tl^|^p|ji^l^ provided a measure of aver-/ 

age growth shown and permitted a rough comparison/ 

of . average -gain sy^le^^^^r^^^^ by different groups of students..- */ 

6.- Was the posttest conspicuously farther from its norm than the pretext 
was from its norm? \ . 



One 6f the primary sep^|3^,ion criteria for participation in Project 
LONGSTEP was 'departure f rj^^adit ional classroom practjlces." It was 
reasonable to expect , therefore , that the average gain shown by a given 
LONGSTEP sample should exceed that of the CTBS norm sample if a greater 
degree of prog ram- leX^el inntivativeness was indeed positively related to 
educational growth/ ■ This question was examined by seeing' if the mean 
posttest score of each analysis sample was notably farther from its norm 
(50th percentile) than the mean pi:^ test was from its norm (50th percen- 
tile). . • ' . \ 

^. It should be noted th^t this method of examining pretest/posttest/ \ 
norm differences was used to facilitate the detection of gross sample \ 
mean/norm differences across grades and years,. This approach was not 
used to evaluate a particular sample's piretest-to-posttest growth with 
respect to national norms. A more appropriate test of such gain in 
achievement is provided by a t test (U.S. Department of' Health, Education 
and Welfare, Offi^ce of Education, 1976), ' • . • 

■ : ' ■■' ^ '■ ' 

; ' ' Y - Y • . 



S^ + S"" - 2r S S 

-JL_ y xy X y (II^l) 

N ~ 1 
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where, _ 

Y- = observed mean posttest* score 

Y = expected mean posttest score (estimated from 
national norms on the basis of the percentile 
rank of the pretest mean) 

= pretest standard deviation ■ ' • 

f posttfest standard deviation 
r = correlatiron between pretest and posttest scores 
N = nunnber of children 
N-1 = degrees of freedom'. 

7. Can^the different analysis samples be distinguished iit terms of the . 
Level of Innovatd:on or Number of Minutes per Day to which they have been 
exposed? 

If achievement growth differences among analysis samples were present 
to. any notable degree, the treatment attributes of the group-s were compared 
to see if the groups were, on t,hfe average, exposed to substantially differ- 
ent kinds of treatments in terms of emphasis on innovation. 

: 8,. Wh^t growth model was examined? • \ 

The model of achievement growth analyzed for this supplementary 
report was the same as that used for Cohorts 1, 4 and 6 and discussed in 
Volume I. Posttest achievement was hypothesized to be a function of 
four sets of predictors: 

• student Socioeconomic Status 

• initial achievement status or pretest ' . . 

• , Level of Innovation and Number of Minutes per Day 

(of class time spent on language arts or math) * 

• Teaching Qualifications. s 

. Regression ^palysis and commonality analysis were used to examine the 
relationship^^ of these predictors with each other (as joint/confounded 
predictors of postte^rt) and with posttest. , 
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9. Were the educational experiences -of students who achieved substan- 
tially more than expected for two consecutive school years notably 
different from thosfe .of students who achieved substantially Xess than 
expected? . , ' 

The overall analyses based' on all students "may ' have shown that j aver- 
age group differences in achievement growth were jio£ highly associated 
with intensity of innovative emphasis. It was possible, however, that 
some students did show dramatic 'gains in achievement and that they tended 
to be. exposed to educational environments which differed systematically 
from the environments of students whose performance was substantially less 
than expected. In short, substantial associations between gains and 
achievement may have been masked in the overall analyses by the majority 
of students for whom the different treatments U^d ^no notable differential . 
dmpact. For this reason, it wa s^f interest to see if dramatic aehieve- 
ment gains or lack of gains on the part of individual- students tended to. 
occur in substtotiallyi different educational Environments. ^ 

Students in each analysi?s sample -whose posttest residual score ^ 
(residualized on the. basis of pyre tesf, and SES) was equal to or greater 
than one-half of a standard delation from the- mean of the residuals 
wei;e identified. The SP72 and SP73 samples for each cohort were merged, '^'^ 
and the students who 'achieved more than expected during both school years ' 
(i.e.,' the '^pAsitive outliers" or "overachievers") and the students who 
achieved less than expected (i.e., the "negative outliers" or "under- 
achievers } were selected for analysis. A dummy variable encoding group . / 
^membership was .then assigned to each student (overach'ievement? = 2; under- 
achievement = 1), and this dummy variable was correlated with the various 
key analysis variables. The ' resulting point-biserial correlations provided 
an index, of the mean difference between the groups on each key analysis 
variable. The correlations were then examined to see if overachievement 
or underachieveirient in two consecutive school years was highly assdciated 
with attributes of the educational treatments to which sych students had 
beeji exposed. As with similar analyses described in Volume I, significance 
tests were not performed because of the highly select nature of subs^^les 
analyzed. ' < ; . 

11 
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10. What approach was used to examine overal l trends in the ; 
data? • ; 

••' -'^ ' • 

To facilitate examining the^^ata for overall trends, the results 
based upon a given statistical te^nique vere summarized in cohort-by- 
grade matrix form like the table of posttest means shown in Table 3. 
(.The statistical analyses summarized>^in these cohort-by-grade tables have 
not been included in the body, of the report — s^e Attachment A for the 
location of these complete tables.) Examination of Table 3 shows the 
following: - , 

• The results or descriptive statistics present in two 
adjacent, cells in the s^m^ row pertain to the performance 
of the ^ame cohort of students, but .during two consecutive 

_ ' school years , 1971-72 and 1972-733. 

• The results or descriptive statistics present in -adjacent 
•cells ill the same column pertain to the performance of 

' different cohorts' of students during the same school year . 

' • Only the two cells along the major diagonal are filled 

because Project LONGSTEP collected posttest data during ■/ 
only two school^ years, 1971-72 and 1972-73, 

This particular manner of sufemarizing the findings was utilized 
because it enabled us to examine and compare the performance of a given 
cohort across school years and grade levels and to relate this performance 
to changes in school environments in different school y^rs. Such ah 
approach may.be viewed as ^an app*roximation to a "within-^ubjects" longi- 
tuilinar analysis since/^the student composition of the two analysis samples 
(1971-72, 197Z-73) for a given cohort was fairly similar (see footnote 
below) . • • * ■ \ 



^ To makfe maximum use of the data available, all student^ present 
during Year 2 (1971-72) or during Year 3 (1972-73) who had a valid pre- 
test score from the previous school year and no missing data on the key 
analysis variable were included in a given analysis sample. Therefore, 

the student composition o'f these samples could vary somewhat. 
>v . ■. 
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TABLE 3 



Posttest Means and Standard Deviations, - Reading Total Score Analyses 





1 


* 

1 • 2 


3 


4 
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634.62 
83.86 


683.08 
85.80 
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682.19 
82.82 


707.35 
92.52 




Furthermore, this mode of reviewing .the study 's results also permits 
a between-subjec;ts**^form of 'analysis in which tJ^e^ performance, of differ- 
ent* student groups, ^t the same grade ' (but during different school years) 
can be compared and related to changes tn school environments. Since ' 
adjacent cohorts of students attended approximately the same schools, the 

• approach u^ed should^e helpful in determining whether q^r- not treatment 
impact is unique to a given group .of student? and/or school year. 

^ ? 

V. RESULTS AND DISCUSSION 

• Review 'of Objectives and Approach 

The cfverall objective of this supplement to Volume I of the Project 
LONGSTEP ^f inal report has been to assess trends in achievement growth, 
ac^rpss grades, ^cross groups/cohorts of students, and across the study's 
.•two primary cognitive outcomes7-reading and arithmetic achievement. The 
specific analytic , objectives developa'd to achieve this overall goal were 

• to determine,, across 'all grades and school«>years, if the * 
average ppsttest^reading aiid mathematics performance 

, ' \ present in. LONGSTEP ' s sample of 'fairly innovative. schools 
. . '\ was conspicuously farther from national norms than wa^^' 
the samples' average pretest scores 

^ ■ .. ^ 

• to determine, across all grades, and school years, if 

' variation a^ongand within analysis samples with respect 
to growth in reading artd mathematics achievement was 
•positively associated with vajjdation in program-level 
i emphas is on innovat ion ^ 

• . ^ ■ ' \ 

• to det^^rmine if . those individual students who achieved more 
than expected (on the basis of their SES and previous 

' year's achievement level) during two consecutive school years 
were exposed to programs with' greater emphasis on innova- 

• ' tion than were students who achieved less th^n expected. 
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' Findings for Read ing Achievement 

w;— ^ 

Average achievement di fferences ambng arialysis samples . Results of 
-the reading achievem^t analyses, showed that the averalge posttest per- 
formance (see- Table 3) at each grade level was' fairly similar, even when 
different groups (i.e., cohorts) of students were involved during consecu- 
tive school years. For example, the mean posttest Reading Total Score 
(Table 3) for different gruups of sixth-graders was 507 (for students in" 
Cohort 4 during 1973-73) and 505 (for students in Cohort 5 during the 
1971-72 school yea^. As shown in Figure .1, with the possible exception 
of growth during the third grade'(i.e., between grades -2,7 and 3.7) and ■ - 
during the eighth grade, average posttest scores for the Proje.ct LONGSTEP 
analysis samples were not farther from national norms than were their aver- 
age pretest^ scores. Although average pretest'_and posttdst ' redding achieve- 
ment for all analysis samples were above national norms (except for Cohort 
5 during the seventh, grade), the reading gains shown by the set of fairly 
.innovative schools participating in the study did not notably exceed the 

growth expected on the basis ,of CTBS norms.'* 
f ' 

Table 4 contains the pretest^to-posttest gains for each analysis 
sample. Inspection of thpse results suggests that -the largest gains in 
reading seemed to have occurred in the' elementary grades' The following 
"irregularities" also seem to be appaTent. ^ 

• The average gain shown by students in Cohort 5 during the 
19 71-72 school year (when they were sixth-graders) was 
notably larger thai! their' gaia the following school year 
(when they were seventh-graders). ^ . 

m Cohorts 6 and 7 demonstrated different amounts of grovS 
(on the average) during the eighth grade, , but Cohort 7's 
average^ gain decreased somewhat during grade 9. 

/ 



CTBS pretest .and posttest national norms, for Reading Tot^l Scores 
and the number of students in each analysis sample are provided in Attach- 
ment B, Tables B-1, B-2 and B-3, respectively. 
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Figure 1. Pretest and palttest means and published national norms (50tb 
percentile for spring testing) - Reading Total Score analyses. . 
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TABLE 4 



Mean Differences Between Reading Total Pretest 

and Posttest Scores and "Standardized Gains''^ 

GRADE, DURING POSTTEST YEAR ' • • ' 

1 2 3^4 5 6 7 8 9 10 11 12 , 



NA^ 


■ NA^ 




/ 






NA^ 


62.20 
.96 




• 




NA2 


45.27 
.71. 






41.38 
.65 


35.49 
.54 




31.60 
.48 





31.6? 
.44 



31.24 
.44 



10.97 
.15 



17.97 
.24 



17.98 
.23' 



41.20 
.56 

0 



26.07 
.34 



29.88 
.39 



30.09 
.37 



14.78 
. .18 



30.70 
.39 



14.13 
.17 



19.23 
.23 



^"Standardized gains" are. pretest/posttc.U mean differences expressed in average or pooled 
standard deviation units—see Volume I, Equation V-1, on page 145. 

. ^llot coiputei because the CTMH was used as the pretest. 



• Average gains during grade 10 were notably larger in Cohort 
8 than in Cohort 9, but Cohort 9's average gain increased^ 
somewhat during grade 11. 

Many of the same results were also evident when average protest and post- 
test reading achievement were compared with CTBS national norms (50th 
percentiles). .Figure 1 shows these ^trends across analysis samples. 

Six possible reasons for such difference among group means were 
briefly considered — group/cohort differences with respect to 

• average pretest level 

• average student Socioeconomic Status 

^ average hevel of Innovation ^ . 

• average Number of. Minutes per Day (of class time on 
language arts activities) 

• average Teaching QiSalif ications ' . . 

• schools attended. ,^ . 
Examination of Table B-4 in Attachment B shows that there were no 

large differences between cohorts with respect to average pretest leved at 
the same grade level. Neither were there large group differences with''''^ 
respect to average SES (see Attachment B, Table B-5) . Although there were 
differences with .respect tq mean Level of Innovation (Attachment B, Table 
B-6)' and Teaching Qualifications (Attachment B, Table B--7) to which the 
various analysis samples were exposed, brief examination did not suggest 
that any consistent relationship existed between these mean attributes and 
the irregularities in average gains noted previously. 

. Except for the lack of hypothesized, conspicuous ^positive growth in 
achievement, the most systematic relationship across the various Project' 
LONGSTEP analysis samples seems to be between average Number pf Minutes 
per Day (of class time on language arts activities) and average ''achieve- 
ment gains. Table 5 shows that students in grades J. through 6 were 
.exposed to about one and a half hours per day (on the average) of class 
activities devoted to language arts. Beginning with the seventh grade,, 
however, students who were enrolled in "language arts"^ kinds of classes 
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TABLE 5 ' * 

Number of Minutes" per Day Means and Standard Deviations - Reading Total Score Analyses 



. GRADE .DURING POSmST YEAR 

1 ; 2 3 ',V ^ 6 '7 8 9 10 . 11 12 
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were exposed to less than one hour of language arts per day — about one 
45-ininute 'to Sb-mijiute class period per day. Since there also seems to be 
a drop in mean achievement gains betweei) the sixth and seventh grades, it 
is tempting to speculate that less exposure to instructiqn (at least as 
measured by the index Number of Minutes per Day), at the junior high and 
senior high scihool levels may be one/ of the reasoniS the gains demonstrated 



jne/ < 

by these Project LONGSTEP grades wete less than the average gains shown by 
our^ elementary grades. However, a number of possible alternative explana- 
tions for" this trend should be considered. 

Perhaps one of the most immediate explanations for the average read- 
ing aqhieyeltnent gains- to be less at' the junior high school and senior high 
schpol levels is that t]ie students who were enrolled in language arts 
classes in these grades may have tended to be in remedial classes, espe- 
cially in the last two years of high school. This is not a completely 
reasonable explanation :^r the trends observed here, however, because all 
Project LONGSTEP analysis sample pretest means were above national norms 
(see Figure 1). Since the students included in each anaj,ysis sample were 
only those who were receiving language arts instruction (i.e., they had no 
missing data on the Level of innovation index or the Teaching Qualifica- 
tion's of their language arts teachei;s) , it is more correct to assert that 
all LONGSTEP analysis samples, on the average, were comprised of abqve 
average readers (relative to the 50t'h percent ile norm) . In fact. Figure 1 
even suggests that students in the junior and senior high school grades, ^ 
on the average, had slightly higher pretest scores (relative to norms) 
than did students in the elementary grades. ^ 

A second possible alternative explanation for reading gains to be less 
in the higher ^ades is that the elementary schools participating in Project 
LONGSTEP were nQtably more effective than were the junior/senior high ^ 
schools. As. explained more fully in Volume I, the particular grades 
selected and tested at a given site/school depended on the specific grades 
at which the site's special program(s) were targeted. Table B-8 in , 
Attachment B shows the school location for all cohorts of students during. 
Year 2 and Year 3 of the study. Examination of -this table shows that 
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• the schools in which members of the same cohort could have^ 
been enrolled were not always the same during the 19 71-72 
and 1972-73 school years " ^ ' 

• the school membership of , the cohorts differed somewhat, 
.^^specially in consecutive school years. 

Thus, some of the small fluctuations in growth in average reading achieve- 
ment among analysis samples • could have been due to the fact that the^^^stu- 
dents in two analysis samples> (either the' s|me cohort during. two consecu- 
. tive school years and at two different V.ade levels or different cohorts 
during two. consecutive school years tut at' the same grade level) were mem- 
bers of a somewhat different set of schools'. Adjacent cohorts, however, 
tended to be members o'f the same or almost the same schools. :. 

Another possible Explanation is that language arts activities in the 
elementary grades may be Qualitatively different from such activities in 
junior/senior high school .y 'Language arts" dufing the elementary grade 
years undoubtedly involves pVimary emphasis on reading 'skills, whereas 
"language arts'.' in junior/senior high schools usually involves a much 
broader set of elective courses\or "enrichment" activities. Thus, instruc- 
tional or class time differences X^etween, the high school an'd elementary 
grades could be confounded with a Curriculum Content dimension, and it is f 
this factor that may be the crucialXvariable . , 

Fin^LLly, it is also possible that the overall trend for reading gains 
. to decrease, during the later grades maXbe due, in part, to (1) the par- 
.ticular manner in which the CTBS ReadingXxotal Scores wera scaled (Expanded 
Scale Scores provided by the test publisheV were used) and/or (2) "the 
tendency for the acquisition of reading ski\ls to' follow the <:lassical 
learning curve. Since exposure to language ^ts activities also decreased 
in the later grades, it is possible that the observed across-analysis- ' 
sample association between growth in reading achievement and Number of. 
Minutes per Day may be merely a result of concomitant variation. 

ler possible explanations f^r the concomitant 
?erage reading and in exposure to language • arts 
itriguing-,to note the presence of such an across- 
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analysis-sample trend in the Project LONGSTEP data.* The fact that this 
loss in rate of cognitive growth was most ^dramatically demonstrated ;by a 
fairly constant group of students (Cohort 5) during the years they moved 
^ f rom elementary to junior high school cer.tainly suggests that changes in 
reading curricula and -school membership^ as well as in instructional time, 
may be important factors to consider in optimizing, reading achievement 
growth in .»|:he. higher grades.' The following section examines the relation- 
ship of Level of- Innovation, Number of Minutes per' Day and Teaching 
^Qualifications to achievement within each analysis sample. 

Associations with achievement within analysis samples .- Commonality 
^ analyses (see Attachment B, Tables B-9, B-10, B-11 and B-12) of the four 
sets of predictors in the growth model (pretest, SES, Level of Innovation 
and Number of Minutes per Day, arid Teaching Qualifications) showed t^iat 
very little posttest variance was uniquely associated with eithejr of the 
two sets of^ treatment measures — Level of Innovation and Number of Minutes 
per Day (as a set) and Teaching ' Qualifications . Ii; fact, in only one analy 
sis sample (Cohort 1, SP73) was the uniqueness for either set of treat- 
ment variables larger 'than* .02. In this one sample, on the other hand, 
Almost nine percent of the variance in reading achieven^nt posttest scores 
for third graders (for Cohort 1 during the 1972-73 school year) could'be' - 
attributed uniquely to Level of Innovation and Number of Minutes per Day. 
^Nevertheless, , the uniquenesses shdy that there ,w as no general trend within 
analysis samples (i^e., involving different grades and , cohorts/ groups of' 
students) for there to be substant.ial\line^r associations between growth 
in reading achievement and unique variation in Project LONGSTEP^s' primary 
treatment indices, i.e., program-level emphasis on innovation (Level of 
Innovation) and typical class time spent on language arts activities 
(Number of Minutes per Day). ' .' / ^ 

«V' . , 

Examination of the growth model regression^analy ses-'lai^^a^ sho^flfed that 
neither Level of Innovat ion no r N umb e r of Min u tes p er Day t en de d to 
receive statistically significant coefficients that..were of the same si^ 
across analysis samples (i.e., across different grades and cohorts) . 
These results are summarized in Tables 6 and 7. Therefore, program-level 
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■ TABLE 6' , 

Sign' and Statistical Significance of the Level of Innovation 
Regression Coefficients - Reading. Tytal Score Analyses ' 
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TABLI 7. 



Sign M Statistical Significance of the Number of Minutes per Day 
Regression Coefficients - Readingiiotal Score Analyses 
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emphasis on innovation neither accounted for substantial amounts of post- 
test variance within 'this set of analysis samples nor did it have small 
associations that were consistently positive across ^different analysis 
data sets (i.e., across grades and gr^^ups/cohorts of students). - 

Table B-12 in Attachment B shows that the unique component of Project 
LONGSTEP's teacher index, Teaching Qualifications, was not highly asso- 
ciated with post test reading achievement; that is, it accounted for almost 
no post test variation independently of pretest, SES, Level of Innovation 
and Number of Minutes per Day. Table 8 shows that the regression coeffi- 
cients for Teaching Qualifications in our growth model also showed no 
obvious tendency to be statistical^ significant and of the same sign 
across analyses. 

Lack of consistent relationships across grades and cohorts/groups of 
students was also shown in the n:orrelations of Level of Innovation, Number 
of Minut«& per Day and Teaching Qualifications with a posttest score 
residualized ob the basis of pretest and SES (technically, a residualized 
growth/gain score). Tables 9, 10 and 11 summarize these correlations. 
The within-analysis-sample trends, then, also have not shown that program- 
level emphasis on innovation was highly or consistently related to growth 
in reading achievement in any general manner. 

As noted in the discussion of methods in Section II and in Volume I, 
it is possible that program-level ^ftiphasis on innovation may not be highly 
associatiBd with achievement since it may not have large average effects — 
that is, it may not be substantially related to achievement for t'he major- 
ity of students in a particular sample. It is certainly possible, however, 
that the more innovative approaches sampled by Project LONGSTEP may have 
been extremely important for some ^ students. To detect such associations, 
those students wh^achieved more than expected on the basis of their par- 
ticular pretest and SES scores (during two consecutive school years) were 
identified. A comparison group was formed from those students who showed 
notable lack jafi^g row th during two consecutive school years. The numbers 
of stud^rit^Vj^aentif led and compared were not- large (see Table B-13 in 
AttachmervqWB)'. Nevertheless^, the **qverachievers** and ''underachievers" 
differed .JrWLatically with respect to average reading achievement during 
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TABLE 8 

Sign and Statistical Significance of the teaching Qualifications 
■ Regression Coefficients - Reading Total Score Analyses ' 

GRADE DURING POSTTEST YEAR ' 





1 


I 


3 


4 


5 


'. 6,, 


7 


8 

V,'. 


9 


10 


11 


12 


0 


NA 


+ 
NS 






















1 




. NS, 


+ 

.001 




















2 




+ 

.001 


NS 


1 
















3 

/ 




r . 


+ 

.05 


.03 




















NS 


+ 
NS 














0 5 

X 

0 
0 


J 


+ 
NS 


.01 












• 6 






NS 


■ + 

NS 

t 










1 


i' 


NS 


.05 




I 




8 


1 


.001 


NS 






* 






+ 

NS ' 




10 




V 


+ 
NS 


+ 

.05 



TABLE 9 

Correlations Between a Residualized Posttest Score dnd 
Level of Innovation - Reading To^al Score Analyses 
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TABLE 10' 

Correlations Between a ResidualizedlPosttest Score and ^ 
Number of Minutes per Day - Reading Total Score Analyses 
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GRADE DURING POSTTEST YEAR 
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TABLE 11 

Correlations Between a Kesidualized Posttest Score 
and Teaching Qualifications - leading Total Score Analyses 
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Year 3 of 'Project LONGSTEP and certainly represent students^'wlio, for two . 

* ■ * * 

years in a row, did either much better or much worse than their achieve- 
ment level or home background would predict. A point-biserial correlation 
between overachiever/underachiever group membership and Year^'l test per- 
formance was computed to provide a measure of the relatJ.ve difference 
between the m^ans^or these -'two groups. Table B-14 in Attachment B shows 
that the overachievers did, on the average, tend to have somewhat higher ^ 
Year 1 test scores than did the underachievers. SES dif ferefi^ces among, 
these two groups of students were not so notable.. Table B-15 in Attachment 
B shows that in only two cohorts/groups of students was there an SES dif- 
ference among achievement groups worthy of-xnote (Cohorts 3 and 5)/ l.n 
these two groups, overachievement was. associated with higher SES l^ve^^ 
(relative to these overathieying and underachieving sti^ents, at least) . 

>Mean dif f erences ■ ^rf the. Jfear 2 and especially the Year " 3 test- scores 
^f the overachievers arid ' uncj^rachievers, however, were dramatic (see Xable 
B.-16"in Attachipent; B)>V/Th'e- magnitude of difference, as indexed by the. 
point-biserial correlations, seems to suggest that consistent overaofii^^j/e- ' 
ment and underachievement , at least *as defined here, is not- highly asso-\ 
ciated with initial achievement status or SES across cohorts. ' • 



Point-biserial correlations between overachiever/underachiever group 
membership and the treatment variables and teacher inde'x in the growth 
model show that underachievement or oyerachievgment was not highly and con- 
siatently related to either Level of Innovation, Number of J^utes per IJay^,- 
or Teaching Qualifications across all analysis ^pples' (Att ^^^ nt B, 
Tables B-17, B-18 and B-19) T Although there did not appear W^^e ariy dra- 
matic' overall trend across all coharJEs, (ihfe results did' seem tfip auggest the 
following^ relative to uhde t achievers ' > jf^ \. \.'^* ^ 

, ' ' . . ■ ■ ■ ' . -■ -■ 

• Overachievers -whov we rfe in elementary and junior hlg 
Vin Year 1 (Cohorts 1 to 8) were members of prograp 
had a lower Level of Innovation (on the average) dSw 
Year 2 and again during Year 3. ■ " 'V 



% Overachievers who were in high school during- Year 1 (Cx)h6rts 
\. 9 and 10) were members- of t^rograms that had a' highej:: Level 

of Innovation (on the average) during Year 2 and ^gain dur- 

..\ 

' ing Year 3. : 

_ • Overachievers who were in gradesNl to 5 duriafe Year 1 were 

exposed to mone class time onV lanfl|iiage art^s activities than 

• . ^efi 7* . * 

were underachievers during . Year, 2 but less tipe (relative 

to underachievers) in Year 3. . . 

■ ■> ' ■ ^ 

• Ovetachievers in the older cohorts were exposed to. less 
class time on language arts activities during Year 2. ' 

• Overachievers; in all cohorts were exposed to less class f 
time on language 'arts during Year 3. . p 

• Overachievers in manV of the cohorts were exposed to a \ ; 
teacher (or teachers) with; a higher Teaching Qualifications » ' 

. ^' score during at least one of t,he two posttes-t school years 

(i.e., either Yeat 2 or Year 3) - 

Although the within-analysis-sample findings reviewed in this section, 
do not; tend to support th^i hypothesis that greater program— level emphasis 
on innovation will, in general, lead to substantial improvements^ in read- 
ing achievement, tihere were two findings, which seemed particularly worthy 
of note: . , - • 

• the large unique impact of Level of. Innovation and Number 

\ • . * ' . • 

' '.of Minutes per Day for Cohort 1 students during the third 
. , . . ^ ' ' 1^ 

. • the slight tendency across procedures for growth in reading 
achievement during Year 2 to :be positively related to the 
three ^treatment/ teacher indices in the growtj^'tnodel . ^ ' 
(especially Number of Minutes per Day) 'and for Year 3. 
growth to be' negatively related to the same indices.. 

Our presentation- of Project LONJi?TEP's findings for reading achievement 
will close with a brief discussion of these two points. * 
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,^ Table B-11 in Attachment B shows that the uniqueness for Level of 

Innovation and Number of Minutes per Day (as a set) accounted for about 

nine percent of the variance in the C^CBS Reading Total Score posttest for 

.-t " . - . 

Cohort 1 students when they were in the third grade (during^- the 1972-73 

school year) . Both predictors also received negative regression coeffl-- 

, cients (see Tables 6 and 7) in the growth model regression analysis of t.hi 

cohort ' ^; data. These same .findings, however, were not observed during the 

previous school year, -1971-72, for Cohort 1 students* during the second 

grade dr for Cohort 2 students during the third grade. Since ^students in 

Cohort 1 and Cohort 2 attended the same schools as second- and third- 

graders, it seems possibsle that the combination of LeVel of Innovation' and 

Number of Minutes per Day was confounded with some set oi jeducational prac 

tices or school personnel factors to wh*Lch the third-graders in, some 

schools were' exposed during the 1972-73 school year and whijph had a no tab 1 

impact on a'fchi^vement . Unfortunately, the ^alyses utilized for this ^ 

report were not designed to "tease out" such effects. 

The apparent trend for growth in reading achievement to be positively 
associated with Number of Minutes per Day' in the 1971-72 posttest analyses 
and negatively associated with the same index in the 19 72-73 posttest 
analyses was considered in light of t-he fact that the Year 2 test was both 
:a;-po.stte*st (in the Year 2 posttest analyses) and a. pretest (in the Year 3 
posttest analyses). It can be shown tfiat under certain circumstances, the 
correlation ^of a 'Variable like Nun^er of Minutes per Day* (NMIN) with a 
Year 2^. posttest residual score will be positive while the correlation of 
that same variable with a Year 3 posttest residual score' will be negative 
(see Attachment C for a short technical discussion). In general, this 
pattern of results could be expected if the slopfe of the regression of 
Year 3 test scores on Year 2 test scores was near 1.0 and the Year 2 
test/NMIN covariance was larger than the covariance of NMIN with the Year 
1 or Year 2 tests. 

The. correlations with Number of Minutes per Day in the Year 2 and 
Year 3 aOAalysis samples for a given cohort, however, show that the Year 2 
test/NMIN correlations in the Year 2 analyses (where the Year 2 test was 



. the posttest) Kere not always similar to the Year 2 test/NMIN corre- 

lations in the Year 3 analyses (where the Year 2 test, was the _^pret^) . , . . 
• Thus, it seems appropriate to assert that the opposite impact "that'iijumber 
of Minutes, per Day had In consecutive school years pi:obably was not an 

- artifact completely due to use of th'e Year 2/test as both.^ posttest and 
as a pretest.^ A nidre appropriate and , reasonable speculation . is that expo- 

:jsure to language arts instruction varied across school years, along with ■ 
changes with respect to the specific students exposed to different amounts 
of*instructipn: in consecutive school years and, with- respect to oChe^ 
attributes of-the program of which . students were memb^s. 

.. vFindings for Arith metic Arhijvempnt ( 

■> . . ^ ■ • .. 1 

Average achievement differences among analysis sampleg . Average 
posttest arithmetic performance was fairly similar for different cohorts/ 
groups^ of students at the same grade level (see. Table 12) during consecu- 
tive schoql years. The oniy exceptions worthy of note'were the mean dif- 
ferences between Cohorts 5 and 6 during the seventh grade and between 
Cohort 6 and 7 during, the eighth grade: Pretest means (see Attachment D, 
Table D-4) appeared to be even more similar across cohorts than were, the 
posttest means. Nevertheless,^ with the possible exception of the growth 
demonstrated fey Cohort 1, students during the third grade- j( in the 19 72-73 
school ar)l average posttest CTBS Arithmetic Total Scores were not 
conspicuously far ther.,f rom national norms (50th percentile) than were 
their average pretest scores. CTBS pre test^ and, posttest means are dis- ' 
played for all samples in Figure 2.^ Since ondf^f the major criteria for 
selectron to participate in Project LONGSTEP .was departure from "tradi- 
tional" classroom practices, these/overall results do not support the 
hypothesis that greater prbgram-level emphasis on innovation is positively 
related to notable gain^ in arithmetic achievement. 

Figure 2, however, does show that the avertage gains in arithmetic ' 
achievement during a given grade level were not always identical for dif- 
. fei?ent cohorts of students. Gains during consecutive school years also 

. ST^e CTBS pretest and posttest national norms for Arithmetic Total -Scores 
Zt n ?T?" of students in e^ach analysis sample are provided in Attach- 
ment D, Tables D-1 , D-2 and D-3, respectively . • , 
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• TABLE 12 /;kv',V 

Posttest Means and Standard .Deviations - Arithmetic To'tal'^o're Analyses 

■ ■ ' ^ T 

■ GRADE DURING POSTTEST YEAR ' J-^"' 
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'Figure. 2.' Pretest ""and posttest means and published national norms (50th 

percentile for spring testing)' - Arithmetic Total Score analyses. 

/ 




were not necessarily identical for the same cohort of students. Across- 
analysis-sampl^ differences were observed both with respect to average 
gains (see Table 13) and with respect to pretest/post.test/norni differences. 

As noted previously, there were no large differences among analysis 
samples with respect to average pretest level (see Attachment D, Table 
D~4) . Thus, the smatll fluctuations in mean arithmetic posttest that were 
observed reflected different medn ^ains in a aunber of analysis samples. 
Inspection of the pretast-to-posttest growth for the various analyses 
samples Csee Table 13) shows that the largest mean gains (relative to 
their standard deviations) seem to be occi«:ring in the earlier gj^ades. 

. In addition. Table 13 also shows the following irregular differences 
among analysis samples: 

• The average arithmetic gain shown by students in Cohort 5 
during the 1971-72 school year (when they were sixth- 
graders) was notably larger than their average gain the 
following school year (when they were seventh-graders); 
however. Cohort 6 students did not demonstrate the same 
achievement losses during the seventh grade. 

• Cohor^ts 6 and 7 showed somewhat different amounts of aver- 
^ age growth di^ring the eighth grade. ^ 

• The average gain in arithmetic performance for students 
in Cohort 8 were larger when they were ninth-graders than 
when they were tenth-graders. 

An examination of Figure 2 also suggested the presence of greater V- 
pretest-to-posttest growth during grade 5 in relation to grade 4 and grade 
6 growth. Project staff examined the analysis samples showing such dif- 
ferences to see if differences among samples were related to similar varia- 
tion with respect to . • 

• average pretest level 

• average .student Socioeconomic Status jgl 

• average Level of Innovation 
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• TABLE 13 

Mean Differences Between Arithmetic Total Pretest 
and Posttest Scores and "Standardized Gains''^ 

GRADE DURING POSTTEST YEAR 
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ISee Volume^I, Equation V-1, p. 145. 
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average Nun±)er iWa^nutes p.er Day (of class time devoted 

':%,:])■':: to arithmetic^,actiiyities) , //* 

■ ' ' ' ''■<i ' 

• average Teaching ^(jualifi-catibn^/ . 

r : ^„ .... ^ \ ^ ' l 

• schools attended.; /' 

Mean differences among -a^alysi-s ^samples with respect to SES, 'Level of 
Innovation, Number of '^^iriu;tes\per Day,\ and Teaching Qualifications (see 
Attachment D, Tables D-S^.^D^e, D-7 arid D-8) did not seem to show any con- 
sistent relationship/with -tlfe irregularities or 
trends noted in Tabi^ 13:'w^.th;respect to -the average arithmetic achieve- 
ment of each analysts .sample^^ For 'example^ the decrease of ^verage growth 
for Cohort 5 stud^tit$ during the/' seventh grade relative to average growth, 
during the sixth'^grade was ngtr, associated with notable mean differences in 
mean SES, Level of. Innovation,: Number of Minutes per Day, or Teaching 
Qualifications. The most likely explanation for the decline for these 
students is that, as. expected,, the .studenfs in Coh6rt ,^5 attpde'd as seventh 
graders schools, different, ftom those the^ 'attended ak^ixth-graders (see 
Attachment B, Table B-g).. it* Seems conceivably that^'ch^^^ not measured 
by tfie primary yariafal^s of interest in this r^^o'rt ptobably .accounted for 
the differences observed.- similarly, the fact.;||^ Cohort 6 students as 
9eventh-graders the year' before did not show thie" same decline^ is probably 
due to the diff erent echdol ' composition of the Cohort 5 and- Gohdl?t .6 . 
seventh-grade /aitalysis samples. * As shown in ^^ttachment' D, Table ;D-^", how- 
^6ver, Cohort; 5 a'eventh-grade students were membe-^ of .EdExAG. groups which, 
on' the average^,; Had a, .great&r pf ogiram-level emphasis on innovation than, 
did the EdExAG groups . for Cohort 5 s^ve^th-graders. / o . 

The dif ferenqes 'in mean arithmetic achievement gains betw^jBti Cohort 6 
and C0hort 7 istudehts ■ during the eighth grade, on the other hand,' showed 
exactly the oppps;t.'te" aysociatloi^ with--,.i nnovat ive emphasi^. 'Althou^ .Stu- '\ 
dents irl toth cohorts were iliembers of the same schools as eighthrgt^ers / / 
(see Attachment B , Table B-8) , greater average gains ( during . 1971V 72 w^^ r 
the .Cohort 7 students were eighth-graders) were associated witii a low'ei: ' V 
p.rogr^m-^leyel emphasis on innovation (on the average) y ^!o«^versely , ► the ; 
difference in average yearly achievement gains during grade 9 'and graele 10 



for students in Cohort 8 was also associatiied (with only a small\ mean dif- 
ference with respect to Level of Innovation and Teach-i^ t5'ualirv;Lcations . 

Greater mean gains for those students was accompanied by a. slightly higher 

" ■> 

mean Level of Innovation apd Teaching Qualif icatif^ns. Mo'i^einent df students 
in Cohort 8 from junior high to senior high schools between the ninffch and 
J;enth grades in some sites could alsq have affected the results Csie 
Attachment B, Table B-8) . Interestingly enough, Cohort 9 students as* 
tenth-graders in 1.97V72 were ^exposed* to much more emphasis on inncAration 
(on the average) than were Cohort 8 students who were in the tenth grade' 
the^ next year (1972-73)— yet, the gains for the two- cohorts during tBnth 
grade were very similar ,(see Figure 2) , 

Thus, the , across-analysis-sample mean differences that*were present 
in Project LOlNGSTEP also do ,not support the speculation that small, but 
consistent, 'arithmetic achievement differences among analysis samples are 
positively related to differences in average program-level emphasis on 
innovation. 



Associations with achievement within analysis sampj.es . Commonality 
analysis fo^ the four sets of predictors in the growth model examined for 
this report showed that Level of Innovation and Number of Minutes per Day 
(as a set) uniquely accounted for more than-one percent of the variance in 
CTBS Arithmetic Total Score posttest variance in only three of 21 analysis 
samples (see Attachment D, Table D-11) , In two of these samples (Cohort 0 - 
SP73 and Cohort 1 - SP73) , the regression coefficients were negative for 
Level of Innovation and positive for Number of Minutes per Day (see Tables 
14 and 15). In fact, the majority of the statistically significant regres- 
sion coefficients for Level of Innovation ilT all analysis samples, taken 
as a whole, were negative. 

About one-half of the statistically *signif icant regression coefficients 
for Number of Minutes per Day were positive. Interestingly, positive 
coefficients tended to be present in Cohorts 0, 1 and, 2 and Cohorts 8, 9 
and 10, Cohox^^ 3 thirough 7 tended to receive negative coefficients. ^ In 
addition, unlike the reading analysis discussed previously, there was no 
apparent trend for the coefficients to be positive- in the 1971-72 analysis 
and negative in the 1972-73 analysis, 
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Sign and Statistical Significance of the Levfei/bf Innovation 
Regression Coefficients - Arithinetic Total' S^ce Analyses 
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TABLE 15 , • ' 

Sign and Statistical Significance of the Number of Minutes per Day ' 
Regression Coefficients - Arithmetic Total Score Analyses 
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Although the ^uniquenesses for T^a*ching QuaMf ications were small in 
all analysis stmples (see Attachment Table D-12), it was notable that 
all but one of the statistic'dlly significant regre3>^ioi:i. for 
this teaclier index were positive . (Table 16). 

Correlations with the residualized posttest score (Tables 17, 18 and 
19)/ also tend' to .show across samples as follows: V 

• Level of ..Innovation, wag primarljLy negatively rq^lated to^ 
the arithrai^tic- posttest 'residual ^score . . *^ , • 

<^ ; %;'ViNiffliber o'f Minutes per Day was positively related to the 
posttest residual scor1| in the young cohorts (Gohot'^s 0, 
1 and 2) and in the oldest cohorts (Cohorts 9 and 10) bi^t 
negatively associated with the posttest residue*, in the 
middle, cohorts. 

'. * ■ ' n? . , ■ 

^ Teaching Qualification^, was primarily positiyely* correlated 

with the arithmetic posttest residual score. . 

■■ ■ " • ■ ^ - . , '.^ 

In review, the regression coe^f ficients and correlations with the 
posttest residual score bot^ suggest\th^*t program-level emphasis on inno- 
vation was not fiighly relate to growth in arithmetic achievement arid that 
the. predomi'hance of the' small associations' ^^sent were primarily negative. 
The tendency for Number of Minutes per Day' Cof typical . class time spent on 
arithmetic/mathematics activities-) to be associated negatively with growth 
in the middle, upper elementary, and junior high school coUorts could not 
be readily explained. ^ . ^ '<y^ „ . " ' 

Comparisons^ o-f st^^dents.who achieved substaritiall^^ore .than expectecf 
^(for two consesjiutive, school years) -with ^student's who achiefVed. less than 
expected show thdt, in general, consisterii^ longitudinal bVeirachieveraerit 
was positively related ta Year latest leyel and SES level (^ee Attachment 
D, Tables D-14 and D-15) . Overachievery undexachiever group differences cp 
these measures, howevBr, were dramatically less than group 'differences-with 
respect to the Year 2 and Year 3 test scores (see * Attachment D, Taljlfe . " 
D-16) . This was a result ,' of *course, of the . procedure used to identify . 
* these particular students. Findings . tended, to p-arallel those just ^ixmmarr 
ized for the regression analyses an^ residual correlations (see Attaclliment 
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SigB and Statistical Significance of the Teaching Qualifications 
Regression Coefficients - Arithmetic. Total Score Analyses " 
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;TABLE 17 



^Correlations Between a Residualiz^d Posttest Score and Level of Innovation, 

Arithmetic Total Score Analyses 
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TABLE 18 , ' , 

Correlations Between' a Residualized Posttest Score and . 
Number of Minutes per D^yj^Mt'hmetic, Total Score Analyse^ * 
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TABLE 19 

Correlations Between a Residualized Post test Score and 
Teaching Qualifications - Arithmetic Total Score Analyses • 
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D, Tables D-17, D-18 and D-19. Relative to the underachievers, over- 
achievers tended, on the average, < 

• to be members of programs with Je^^ emphasis on innovation 
during the 1972-73 school year 4j^i?wp 1 through 8 




• to be members of programs wi£B|^^|^^t'S?!^k^r emp on^ innova- 
tion in Cohorts 9 and 10 ^ 

• to be exposed to less time on arithmetic class activities 
in Cohorts 3 through 8 but more time on arithmetic in 
Cohorts 1 and 2 and Cohorts 9 and 10 

• to be t^^ight by , arithmetic/mathematics teachers Who 
obtained higher scores on Project LONGSTEP's measure of 
'teacher experience. 

The \^"thin-analysis-sample findings for arithmetic achfevement, then 
show that / . « 

• Level of Innovation, in general, was not highly associated 
with arithmetic achievement and that there was a somewhat 

; consistent 'tendency for slightly greater arithmetic ^ 

achievement to be associ-ated if^^M^ less program-level ^ 
emphasis on innovation 

. • Number of Minutes per Day was positively related to 

arithmetic achievement but .only during early elementary 

• ■ ."school and late high school 

growth in arithmetic achievement was positively associated 
with Teaching Qualifications. 

Overview of Trends for Reading and Arithmetic Achievement 

The purpose of the analyses conducted far this supplement to Volume 
of the Project LONGSTJ]? final repart was (1) to ascertain if the trends 
observed and discussed in Volume I with respect to 'Cohorts 1, 4 and 6 were 
y representative of the trends shown by all analysis samples and (2) to com- 
7pare results and determine if other meaningful trends across cohorts were 
present. The overal'l findings reported here have shown the following: 
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The mean reading and arithmetic posttesfc scores for Project 
LONGSTEP's sample of fairly innovative schools were ftot 
conspicuously farther Irom national norms than their -average, 
^pretest scores were from their norms. . " , 

• Variations among analysis samples v^th_ respect to average 
reading and arithmetic achievement /gains di'd not tend to be 
^ associated in.any highly consistent manner with concomitant ' 
. differences in the mean Level of Innovation, Ntftnber of 



Minutes per^Day, and Teaching Qualifications of the s'amples . 

• Variation in Level of Innovation was not highly associated 

with reading or arithmetic achievement within Project LONGSTEP's 
analysis samples. . ' ' > ' 

< ■■ 1 • 

• Variation in Level of Innovation was not consistently or 

positively related to reading achievement within analysis 
samples » 

• Variation in Level of Innovation appeared to be negatively 
associated with arithmetic achievement. 

• Teaching Qualifications was not highly or consistently 

■ , ^elated to reading achievement but was positively associated 
with small gains in arithmetic achievement 

'Additional findings included (1) a trend for reading and arithmetic 
achievement to decrease between the sixth and seventh grades , (2) a trend 
for reading" and arithmetic gains to be larger in the earlier grades, 
(3) a trend for the average gains in arithmetic shown by all ^cohorts 
except- those in senior high to be larger than mean gains in reading 
(relative to their respective standard deviations), (4) a trend for "the 
elementary graces (1-6) to be exposed to notably more instruction per day 
than junior high and high school students with respect tc5' language arts, 
and (5) notable^ mean achievement gains in reading and arithmetic foT 
'students who were third-graders during the 1972-73 school year. 
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In summary, the primary research hypothesis, that substaiAlal gains 
in cognitive achievement are positively associated with innovative 
'emphasis, has not been supported in any general way by the analysis of 
Project LONGSTEP's data. These results, based on a. global analysis of 
trends across reading and arithmetic analysis samples and cohortsAgrades 
tend to confirm the findings reported in Volume 1 for Cohorts 1, 4 and 6 



49 



REFERENCES 



'Coles, G. J., Chalupsky, A. B., Everett, B. E.", Shaycoft, M. F,, 
^. Rodabaugh, B\ J., & Danoff , M.'N. ' Xmpact of educational innovation 
on student performance: Project inethods and findings for three 
cohorts . Project LONGSTEP final report: Volume I. Palo Alto, 
California: American Institutes for Research, 19 76. 

U. S. Department of Health, Education, and' Welfare, Office of Education. 
A procedural guide for validating achievement gains jLn educational 
projects . Washington, 'D. C: U\ S. GSvemraent Printing Office,^ 
' 1976- 



50 

77 



0 



■ I 



ATTACHMENTS 



78 



ERIC 



ATTACHMENT A / 

. . / 1 ^. ; ••' 

Loq^^'tion of the Complete ^jStatisti.cal Tables^.- V* 

Summarized in Volume I Supplement > ^-i ^ " » ' 

■ 4 ' - ■■ . ■ ' " ■ ■ / • " . ' • 

Analytic methods implenpnted specifically for the Volume I Sirppleifien.t- 
involved the data for Cohorts 0, 2, .3, 5, 7, 8, 9, aAd 10 for two cognitive 
* dependent variables, the CTBS Reading Total and Arithmetic Total Scores. 
The focus of this report, . however, .was not on detailed comparisons of the 
results shown by individual cohorts but on the overall conqlusions based ^" 
on findings from all cohorts (including Cohorts 1^ 4, and 6 .fiiscussed -in • 
Volume I). . Thus, the Gohort-by-grade tables iticdrporated in the Volume I 
Supplement summarize results obtained either in /these analyses or in 
.those- conducted for Volume I itself. Table A-l' shows thejlocation of /the 
statistical analysis tables for each cohort.. .The content 'oS the Volume I \ 
Supplement Appendices is *sh*own in Table A-2 . • 



Table^-A-l 

Volume^ Location''' of All 
Statistical* Analysis Tables 



Cohort 



Volume I or _ Volume I 



-0- . . . 
1 

2 


. X 




, : ■ -O.:-^^ W.-* ^ 

-■>.., 

* ■ ■ , X " ■ . 


• 3 ■ 










X 














6 . 


*x 

* 






7 - 






■ ■ ' "•■ ■■' A' X • ■■■■ 


; 8 .• ■ 








• . "■.■V 
10 ' 

4/ 






.■' / r: - J ; A '. ■ . 

■ ■ x • 



c 



Volume I, the Volume I Appendices, anS the Volume I Supple- 
ment Appendices are each separately bound documents and 
will be deposited in ERIC at the' c^dliision of the project. 



a-l 



TABLE A-2 



Corvtents of the VoluAe I 
Supplement Appendices ''^ 



Appendix Content 



All Reading Analysis Tables for Cohorts Q, 2, 3, 
^^"7, 8, 9, and 10 / • 



Tables 

.:' . •'A-i ,tp. A-15 ■ Means and Standard Deviations 



; ;: -^./^c'^ V B-;.!' to* B-15 , Intercorrelations 



^ ''x *' . - f^\ \t'^.' aI:^*^^ C-15 Commonality Analysis Tables 



S*""^ - D-1 to D-2 Regression Analysis Results 

All Arithmetic Analysis Tables for Cohorts Q, 2, 
3, 5, 7, 8, 9 > and IQ ' o " 

Tables 



^ A-lrto A-15 , Means and Standard Deviations 

^4^ B-1 to B-15 Intercorrelations 

C-1 to C-15 ^ Commonality Analysis Tables 

D-1 to D-2 Regression Analysis Results 



Project LONGSTEP Profile Variables and Variable 
Abb re viat ion Summary 
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' . ~ • ATTACHMENT B * 

Supporting Tables « Reading Total Score Analyses 

■ ^ 
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The first box in each row contains results for the Spring 
1972 posttest analysis; the second box contains the Spring 1973 
posttest analysis results. ^ In boxes containing two values, 
the first statistic described in the table title is the upper 
value and ^the -second statistic described is the Icjwer value. 
Cells along the diagonal for which data were "not available" 
or "not applicable" contain an **NA." The abbreviation "NS" 
has been used for "non-signif icanj^ " 
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TABLE' B-1 

Pretest national Sons (5S.fi Percentile) - Reading lotal Score Analyses ■■ 



>5 



GRADE DORING POSnEST YEAR 
■5 6. " 7 8" • 9 



10 



i 

11 




TABLE ,B-2 , 

Posttest National Norms (SOth.J'ercentile) -'l^eading Total Score Analyses 



^ 1 


ff 

' 2 . 


. 3. . 


4 


G'RADEJURING POSTTEST YEAR ■ 
, . 5 ' 6 ■ 7 8 


9, 




' li 


12' 


na' 


322 ■ 

If . 














1 










321:5^ 


375;5 










I 








f 




375.5 


419 


ft, 




\ 














u 


457.0 




\ 

% 




b 






f 

1 


' . i . ' 


457 


49(lv5 






> 


















: 489.0 


: ii 






< 




* 


V 


5 






I. ' 
>* 


V 

% 


526 ' 


"■553.5 




> 








'"i 






— 






555.5 


'586 




i 


1 


-Hi- i 

— ^ 




1 

■ ^ ■, 


r 

, *♦ . 




i J 






CQQ 


oio 
























618 


U 

1 


















1 






NA 


. NA 




0. 



4 ■ 



I i 



4„ 



"0 ■ ■ 



■V. 



7;' 



.9 



10: 



u ' . *' :. TABLE B-3 
^ ;Nuiiiber of Students by Analysis Sample - Reading J|i|l Score Analyses:.;;,; 
■ ■ ^ GRADE Mm POSTTEST YEAR'.^,# ' 



8ERLC 



6)4 



791 



10' 



ii:;.;': ily/v.;;;.;/- 



:J67 



805 



75i 



1879;*. 



652 



J- 



A 



0— 



■'t?25,.,.. 



' V';' 

16 



179,4 ^' 



-r- 

"'V 



:^i48f: 



■ p 



^241 

"J 



.r:i406 



'V. 



1238. 



'Mr: 

M 



SI 
.... 



>■ I . 



... - 



■ V 





•■ ¥ ■ 


















'■■.'• 






J: 
















Pretest Means and Standard Deviations - Reading T(M Score Analyses 



GRADE mmQ POSTTEST YEAR 
5 1*6 . 7. 



2 ' 3 4 m ' ' 1: 8 9' 10 



11 



12 



NA 



92.44' 
11.00 



93.85* 
10.56 



33$.46 
63.87 



105.28' 
11.36 



386.11 
63.49 



390.04 
63.72 



435.07 
60.56 



437.91 
64.-25 



475.73 
67.30 



473.°82 
71.00 



512.31 
65.08 



513.30 
72.78 



545.45' 
72.79 



538.00 
72.43 



580.83 
70.68 




r 



Based on the CTMM. 



TABLE B-5 



■1 



. Socioetonomic Stat'us-lSES) Mean3 'and Standard Deviations : Reading Total; Score Analyses. 



2' 



GRADE DURING POSTTEST YE'AR 
f 6 ■ 7 ' 8 



10 



NA 



> 



10 



100.72 
10.1? 



100. Oj 100.59 



10". 02: 



9.94 



99.88 
9.99 



99.97 
9.70. 



99.54 
9.75 



100.27 
9.99 



99.39 
9.93 



■99.74 
9.90 



98.12 
14.03' 



99.72 
9.73 



99,50 
9.92 



100.08 
9.72 



100.35 

.9. '99 



100.19 
9.97 

100.73' 
10.28 



100.83 
,9.95 

99.37 
9.77 



-4- 



11 



101<t)l' 
10.02 

99.53 

•10.02 



12 



0* 

I ■ 



0 5 

0 



92. 



ERIC 
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TABLE 3-6-^ 

' ■ ♦ 

Level of Innovation Means and Standard •Deviations - Reading Total Score Analyses 



2 



3' ■ « 



GRiiDE DURING POSTTEST YEAR 

i 6 ]■ ,8 



lO- 



NA 



21.25 
3.07 



•JD.12 
4.33 



21.03 
3.26 



"20.12 
4.06 



3.19 



.50 
4.23.' 



19.51 
3.32 



18.77 
4.43 



19:95 
3.55 



19.37 
3.91 



18.35 
4.49 



19.24 

4';74' 



•19.25 
, 4.46, 



19.65 
4.33' 



17.49 
3.61 



18.18 
'3.69 



mi 



18.01 
3.61 



20.15 
4.62 



11 12 



1 



17.74 
3.13 



17J3 
1.5,4 



18.12 
3.39 



: ' leaching Q^llflcations Ms and Standard iSivlatlons - Reading Total S^ore Analyses ■ 



GRADE DORMG.POSIIEST/™ 



0 


NA 


97.87' 
7.55^ 






J 




4 




! 


.V 




A / ' 


1 




97.61 
7.84 


100.90 
6.35 




i 








V" 


y 

s 


♦ 


■2 


i 

f 


97.90 
9.78 


100.53 
7.19 




/ 


« 


; 

1 


■ i 

/ 


f 






3 






101.18 
7.65 


100.67 
8.05 




■ 


- ) 
.J ' 

$' ■ 




■ ' 


! 


I 






102.47 
' ^ 7.88 


99.86. 
7.46 




* s 










5 


V 


100.12 
8.60^ 


98.84 
7^23 






I 






6 




95^26 
/8.24 


99.06 
. 6.53 




\ 


> 




1 




1 


i 










97.47 
9.17 


101.63 
S.U 




1 




8 


1 














100.56 
8.47 


101.01 
7.70 




A 


■9 


/ 


V" 
















99.44 
7.20 


100.72 
. 8.80 




10 


■ : « 

1 '. T- — 


104.74 
,8.13 


103.24 
9.39 



TABLE B-8 



"School Membership by Grade and Cohort^ .■ 



School 

£ 


0 


12 3 


4 


Cohort 
5 6 ' 


7 8 #'-9 


10 




1 

* 




4.5' 

1 


5,6 


6,- ^ 








' 2 - 




4 5 


5 6 


6 - 
















- "7 








4 


\ 




5,6 


6.- . 






B 


5 








-.7- 












■ 4,5 


5,6 


6.- 








« 7 




«^4.5 


5,6 










8^ * 




J 4^5 


5,6 


6.- 








9 






5,6 


6.- 








10 




■ " 4,5' 


5,6' 


6.- 






C . 


11/ 






5,6 


6.- 








12 




.^.5^5.6 


6.- 








13 








-.7 






.„ 


14 








-.7 7,S 


8,9 9^^ 








< 




A • 




-,10 10,11 


11,12 








2.3 3;4 4,5 


5,- 










. 17 

1 Q. 


1.2 


2.3 3.4 4,5 


5.- 
-.6 


C -J 7 O 

by/ / 9 O 






19 












11,12 




zu 

21 




4,5 


5.6 


-.7 








22 




/^^4,5 


5.6 


6,- . 








23 




4,5 


5.6 


6,- 








24 ^ 




• ,- .. 4,5 


5.6 


O 9 

-,7 7,8 


a 

8,- 




F 


,26 ' 










-,9 9,10 ■ 






11 




4,-2 












28 




4,5 


5.- 






* 




29 

30 y 




4,-2 










G 


31 




V, 


-.6" 


6,7 7,8 


8.- 

-.9 9.10 

^ — <ii II ''i ■ 





















irl'tinaed) 



TABLE B-8 (continued) 



School 


0 


1 


2 , 


3 


4 


Cohort . 
5 6 


7 


8 


^ 

9 


10 


Site , 


33 : 








4-, 5 


5,- 






/ 










.34 


1,2 


2.3 


3.4' 




5.6 


6.- 














35 , 

i' 


t ■ 








-.6 


7 

_ 7 
1 ' 








* 




H 


37 








4,5 


5.6 


6.- 




^ 










38 








4,5 


5.6 


'6'.- 














39 












■J 


7 Q • 


8,9 


9.- 






I 


. 40 


















-.10 


10.11 


11.12 




.'41 • 


1.2 


2'.3 


3. A 


4,5 


5,6' 


6,- 














42 


1.2 


. 2,3 


3.4 


4,5 


5.6 


6.- 














43 


1.2 


2.3 


3.4 


4,5 


5'.-6 


6.- 














44 


1.2 


2,3 


3,4 


4.5 


5.6 














J 


45 


o 




.. - 






-. / 






• 








46 • 












-.7 














47 


1,2 


•2.3 


3.'* 


4.5 


5.6 


6,- 














48 


1.2 


2.3 


3.^ 


4.5 


5.6 


6.- 














49 












-.7 


7,8 


8.9 


9.- 








50 ' 












-.7 














51 ' 


















-.10 








52 


1.2 


2.3 


3.4 


4.5 


5.6 


6.- 















53 

55 
56 



1.2 2,3' 3.4 4.5 ,5.6 6,- i/ 
1.2 2.3 3,4 4.5 5, 6 6^-^^^ 
1.2 2.3 3.4 4.5 5r^'^6.- 

-.7, 7.8 8. 



-,9 



58 


4,5 


5,6' 


6,- 




59 


4,5 


5,6 


6,7 , 


■ 


60 


4,5 


5,6 


6 , 7 


M 

/ 


6ll 


4,5 


5,6 


6,- 




.62 






-.7 






Tab'le entries , in each column are the grades for ^:Ke 
(first number) and 1972-73 (second number). • 

^School closed arid students transferred, / 



cohort in 1971 



' b-9 
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• \ , • TABLE B-9, . ' ' 

Uniquenesses and Multiple Correlations (Squared) for Pretest - Reading Total Scor^ ^balySes 

; . • . GRADE DURING POSTTEST YEAR 

1 ' 



r 


, 1 




, 2 


■ ■3 


4 


5 


6 


7 , 


8 


9 


10 


H 




12 / 


0 


• NA 


.139 

.185'; 


















\ 


V 




.1 




.095 
, .134 


.517 






• 












■■ 1 




V 

2'' 


\ 

1. 




.166 


.522 
'.629 
















' • \ 
■ 1 

i 




3 


■,..\ , ^.^ 

I 


.54'8 
.693 


.'494 
.6-64 






1 

t 






( 






4 


\ 


.565 ' 
.703 


578 
.729 








I 




j 




5 




.707 


♦ JDJ 

.680 ' 




* 










6 












< 

t 




« DID 

.738 


.741 




■,.■1" 

1; ' 








1 














9 


i 


, .dUd 
.708 


.688 




r 

■» - ,4' 






oo 






% 


• 












.590 
.694 


.604*.: 
.678' 


' ' '. 




















'.638 
.738 


.599 

;652 

1 






10' 


















1 






;63r 

J 650 

.. 1 ■ . 


.614. ' 
.623 



lABlE B-10 



Uniquenesses '^d Multiple Correlations (Squared) for SES'- Reading Total.-Score Analyses 



NA 



1 



.021 

,.072 



.114 
.164 



.001 

.075: 



.073 
.112 



.008 
J97 



.005 
.14^ 



GRADE I DURING IpSTTEST YEAR 

5. , ■& ' 7 8. 



7 



..013 
.182. 



.1J8 



7 



.006 
.152 

.001 
.062 



.0()4 
.116 
/ 

loT 

.114 



.001 
.118 

.004 
.102 



.001 
.090 

.001 

.104' 



10 ^ 11 



.002 
.085 

"ooT 

.105 



,002 
.045 

.001 
.017 



12 



• Uniquenesses and Multiple Correlations (Squared) * ' 
for LevW of Innovation and 'Nujiber of Minutes per Day - Reading Total 'Score Analyses ' 



GRADE.DURItJG POSTTEST YEAR 
5 • 6 T 



8 



10 



1-1 



12 



NA. 


.op" 

J .009' 

> 


( 




• • 


i 






/ 

r 


J 




4 




.004' 
.008 


m 




\ 


* 

I 


t 


f 


\ 












■.005< 


'.005 
.008 - 


1 1 


■ 




• 


y ^ 

1 




« 


' — 7 






.004> 

,.012'. 


.002 ' 
,009 ■ 




1 


V 

1 






( 


> 

/ ■ • 


■ ' 


-.002^ 
,.002 


.001 

.oi6 












. \ 


< 




/ 1 

> 


• J 




.000 
.004 


.000 
' .021 




/ 








f 

* . * 1. 




.003 
'.045 


.007 , 
.045 " 








it 








f 

r 


V 




5 


.002 
.028 


*010 
.016 








■1, y\'~ 


.■.oa3 

.006 


.021 
.008 


(.„■ 






f 

1 • 




* 






; 

V 


/ 


1 


.008" 
.024 


.016 
.001 




■ .■ ' r . . ■ : 

v' ^ . . ' ^ ' 


■ / 


.001 
.003 


.013 
.018 



TABLE B-12 



■ ■ Uniquenesses kid Multiple Correlations (Squared)^ 
for Teaching Q-ualifications - Rea'ding Total Score Analyses , 



/ GRADE DURING POSTTEST' JEAR^. 
' 5 \ 6 7 8 



10 



\\ ' ■ 12 



■ i 

; ' , .■ 

i 


NA ■ 


M 
.oDo 










4 












, 1 




.002 
.000 


.006 

040 


> 








.1 










7 


4 


.012 . 
.008 ' 


.001 

.000*, 

{l 




1 


■ 










• 


3 


r 

« 




.001 
.000 


.001 ' 
;001 ' 






1 




r 


\ 




H 


ft 


, 1 






.000 
.003 


.000 
.000 




r 

* 


< 








H 

■ i 5 

■ ■ U 

D- 


> 

f 

« 


.000 
.000 

4 


.002 
.012 ' 








i 

) 

1 




? 












„ ,000 , 
.025' 


.000 
.017 






( 

L 


» 

■} 












f 


V 


.000 ■ 

.016 


^.001 . 
".001 








\ 

8 






/ 


t 

; 





1 


01? 


'.000 
.000 






•-9 

M 


"I, ' 

t 


1 






/ ■ 


i' ■' ' ■ *> 






'.090 
.000 


\ooo' 

.017- 




r . 

. 10 




i ■ . . 




* 


J i . 


1 ■ 


• 


I. ^ 


/ 


.oo'o 

.006 


.,.003 
^•,029 ■ 
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I TAB1,E B-13 . ' , , 

Nui)ei*s' of Consistently..Dverachieviag and Underachieving Students Identified 
. ■ Reading Total Score Analyses 

.'(' ■ . 



GRADE DURING POSTTEST YEAR 



1 \' 1;)'/ ) ' 4, 5 



'6 



7 



9 



'1 



Ml . ■, 

(« X 

, 0' • ■ 



m 



ERIC 



10 



NA 



• ■ 57 



• 50 



■ 71 
89 



• 96 
72 



62 
52 



85 
59 



48 
48 



10 



70 
35 



11 



' 31 



12 



37 
17 



0" 



' 3 



h 

•J 

r. 



10 



^ TABLE B-14 

Point-Biserial Correlations Between Student jjutlier GrWp Membership 
and Pretest - Reading Total Scojre Analyses 

GRADE during" POSTTEST^YEAR 5 
6 7 8 • 



12 3 4 5- 


■' HA 










.467 ■ 




' .270 






J30 



9 



10 



11 



12 



,080 



.416 



.134 



.454 



.241 



.253 



' ' TABLE .1-15 ' 

^ A ■ ' r : ' 

Point-BisetiarCorrelations Between-Student Outlier Group Membership 
.. ' ■, ' and SES\;, Reading Total Score 'Analyses 



grade: DURING POSTTEST ^EAR 

5 , .6 1 7"''' ■ ,8. 



0' 



0^ 



■, i 

0 

a 

0. , , 



110 



10 



ERIC 



NA 



.o4o: 



.015' 



-.07 



.352 



-Ml 



■.147' 



I ■ i 



.139 



10' Nil ^ 12 • 



,039 



.051 



TABLE B-16 

Point^iserial' Correlations Between Student Outlier Groiip'Menib^ \ 
and Year 2 arid Teal Mest Score -■Reading Total ^ScokvAnaly^^^ 
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TABLE' B-17 , ' ' 

Point-Biserial Correlations Between Student Outlier Group Membership 
and Level of Innovation - Reading Total Score Analyses . 
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5 



6- 7 



10> 



,11 12 



NA 



NA 



-.170' 



V.438 



-.132 



■Ml 



.017 



-.129 



V 



00 



H 

S 5 

33 
0 
0 



-.161 



-.041 



-.101 



-.116 



-.047 



-.102 



.144 



-.068 



-.145 
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■ ■ ' ■ TABLE B-18 ^ • * , 

Point-Biserial Correlations Betw&n Student Outlier Group Meinbershit) 
. and Nunier of Minutes per Day - Reading'Total Score Analyses, 
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■ ' ^ ' ^ TABlk B-19" 

Point-Biserial ''Correlations Between Student Outlier Group Membership 
and Teaching Qualifications - Reading Total Score Analysers 
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ATTACHMENT C 

Nptes on Correlations with 
Residualized Posttest Scores'^ 



* A.: ^ . ■ ■ 

. •,*;.A t^n^^ was noted across samples for the correlation of Number of 

V J^*^Vit:^a.^ Day during .Year 2 to be positively correlated with the Year 2 

».■■ ..." • ' . ♦ . 

- ^;.readirig posttest residukl score and for Number of Minutes per Day in Year 

's • V ' ' • * 

.;;*3;to be negatively correlated with -thfe Year 3'reading posttest residual. 
.■•i-rProject staff considered -that this seemingly consistent trend in the read- 

.ing analyses m have been an artifact' completely due to use of the 
Z-;^:' ^^^-^ ^ ^^^^ ^® ^^-^^ ^ posttest (in the Year 2 posttest analyses) and as a 

prete^S^^^^^^ analyses). This attachment notes a set of 

• V- circumstances thaV>-c6 lead to, this trend but does not conclude that the 
. . shift iu sign -is y b^^ a statistical artifact. 

: ' V For. fch^^ sake of simplicity , let us restrict our example to a posttest 
. ;:score residual i^ on^^he basis, of a single predictor, pretest. In this 
jcfiisev tKfe .Year 2 post ttest residual can be shown to be equal to- - 



where. 



Y^ = Year . 2 test score- residual , 
Y^ = Year 2 test score, 

a^^ = raw score intercept for the regression of Y^ on Y^, 

= raw score regression 'coefficient for the regression 
' of Y^ on Y^y and 

Y^ = Year 1 test score. 

The cor^(^spoflF5fing residual for the Yeat 3 posttest test Score is, 

= Y^ - - h^^ ' • (C-2) 



1 

Appreciation is e^jpressed to Dr. Robert Linn who provided the 
general formulas upon which this attachment is based. '^^ 
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where , 

^3 ""^ear 3 test score residual, 
= Year 3 test score, 



a^ --raw score intercept for the regression of Y^ on Y^, 

= raw score regression coefficient for the regression 
of Y^ on Y^, and ^ 

Y^ = Year 2 test score. • * ' ^ 

If we now assume that students are exposed l^db tl;^ffeame Number of Minutes, 
per Day (Nl^IN) during both Ye.^V, 2 and Year 3 (an assumption definitely not 

't \' 0 ■■ ■ 

present in the LONGSTEP data), the covariance of the Year 2 residual, Y' , 
with NMIN will be equal to * 

Gov (Y NMIN) - b, • Gov (Y^ , NMIN). (G-3) 

Similarly, the corresponding covariance of NMIN with the Year 3 residual,. 
Y^ , will equal 

Gov (Y^, NMIN) - ' Gov (Y^, NMIN) . (G-4) 

Now, given that Gov (Y^,. NMtN) is the largest of the three covariance 
terms and that b^ is near 1.0, the covariance of NMIN- with the Year 2 
residual (G-3) will be positive and the covarianee with the Year 3 
residual (G-4) will t^e negative. Gonversely, if Gov (Y_, NMIN) is the 
smallest of the three covariances and b^^ is near 1.0, then the covariance 
of NMIN with the ' Year 2 residual (G-3) will be negative and .the covariance 
with the Year 3 residual (G-4) will be positive. Since there' is no par- 
ticular statistical reason for NMIN to be more highly related to the Year 
2 or Year 3 test score and since the Number of Minutes per Day index used 
in these analyses .were gathered during Years 2 and 3 separately and did 
vary bfetween years, the shift in signs observed in the Project LONGSTEP 
residual correlations is • belieyed to be merely a methodological arti- 
fact that is independent the processes being studied. 
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. • . ATTACHMENT D ' 

Supporting Tables - Arithmetic Total Score Analyses 



The first box in each row_ contains results for the Spring' 
1972 posttest analysis; the second box contains the Spring 1973 
posttest analysis results. In boxes containing two values, 
the first; statistic described in the table title is the upper 
value and the second statistic described is the lower' value. 
Cells along the diagonal for which data were "not available" 
or "not applicable" contain an "NA," The , abbreviation "NS" 
has been used for "non-significant," 



120 * : 



a 
I 

H 



, 2 



H 

(Si 

0 5 

0 

u 



10 



TABLE D-1 , 

Pretest National Norms (50th Percentile) - Arithioetic Total Score Analyses 

■ GRADE DURING POSTTEST YEAR 

2' . H 4 5 6 '7 ■ . 8 9 10' 11 
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■; ' ' , ' TABI^ .D-2" 
iQst'test National Norms (50th Percentile) - Arithmetic Total Score Analyses 

■V . Gk)E DURING POSTTEST YEAR 
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, . ' TABLE D-3; 

Number of Students by Analysis Sample - Arithmetic Total Scores Analyses • 

■ . • ' ' GRADE DURING POSTTEST YEAR' ■,, 

'2 3 4 5' 6 ■ 1 "8 . . 9' 10 ■ 11 
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TABLE H 

Pretest Means and Standard Deviations - Arithmetic Total Score' Analyses . 
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TABI,E D-5 

Socioeconomic Status (SES) Means and Standard Deviations - 
Arithmetic Total Score Analyses 

; . GRADE DURING POSTTEST ,YEAR ' 
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TABLE D-6 . , 

Level of Innovation Means and Standard Deviations - Arithmetic TotaL,Score Analyses 

GRADE DURING POSTTEST YEAR 
1 'I . 3 k 5 6 1 8 9 10 , ., 11 ^2 
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Nuiber of Minuses per D^y Means and Standard Deviations - 
Arithmetic Total Score Analyses' 

GRADE DURING POSTTfeST *fEAR 
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' 7 ^ ■ 

, ' ^ ' TABLE D-8^ 

Teaching Qualifications Means and Standard Deviations - Arithmetic Total Score Analyses 
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■Uniquenesses and Multiple Correlations (Squared) 
for Pretest - Arithmetic Total Score Analyses 

, GRA^E DURING POSTTEST YEAR 
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• TABLE D-IO' ' 

Uniquenesses and Multiple Correlations (Squared) 
for SES - Arithme'tic Total Sc'ore Analyses 
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^Uniquenesses and Multiple Correlations (S(|uared) for Level of innovation, 
and Number of Minutes per Day -'Arithmetic Total Score Analyses • ^ 
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Uniquenesses and Multiple Correlations (Sijuared) 
for Teaching Qualifi|ationS', - Atittaetic Total Score Analyses 
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TABLE D-13 

Nuniers of Consistently Overachieving and Underachieving 
Students Identified - Arithmetic Total Score Analyses 
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Point-Biserial Correlations Between Stulfent Outlier Group Meii^ership 
ani Pretest (Year 1) -.ArithmeticVotal Score Analyses 
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Point-Biserial Correlations Between Student Outlier Group Membership 
and SES - Arithmetic Total Score Analyses 
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Table D-16 
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Point-Biserial Correlations* Between Student Outlier Group Mentership 
^ and Year 2 and Year j^Test Score - Arithmetic Total Score Analyses 
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. , , ^ TaUe D-17 

Point-Bisedal Correlations Between Student Outlier Group Menfcership 
' and ' Level of Innovation Arithmetic Total Score Analyses 
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• ' ■ ■ I ' ' , , Table' . 

Point-Biserial {orrelations Between Student Outlier Groiip Membership 
and Number of Minutes per Day - ArithnietiG Total 'Score Analyses 
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Table D-19' 



Point-Blserial Correlations Between Student Outlier Group Membership 
and. Teaching Qualifications - Arithmetic Total Scare Analyses . 



I . 

H 



h 
a: 

0 5 

X 

0 

u 



10 



#.1 


■ 2' 






GRADE 

' 5 


DURING POSTTEST YEAR 

6 ' 7 ' ■ 8 ' 


Q 
7 


in 


n . 


' 12 ■ 


NA 


NA-,. 

V' / 






i- 




I 

il 


6 








V 




A05 


• U/H 












p 










,^;090' 


' 






J, 

» ' 1* ' 
1 


t 








J 
i 






_ 047 ' 

1 w*t / 














\ 






-.049 


m 

.109 












V 


. 1 • •, ■ , 


-.100 


M73 














' .098 ' 


.296 








• 


, 'id 


.263 


-;139' 




■ A . 

0 

* 




' ' ' ( * ' ■ ■ , 


V. 


.125 






: * ' . '• ■ ■ ^ 

i 


.043 


-.161 




f ■ 


f 












■r— ' 






. .019 

> 


■ ,165 



.5,. k. 



1" O 



